\documentclass{article}[12pt]
\usepackage[a4paper,bindingoffset=0in,left=1in,right=1in,top=1in,bottom=1in,footskip=.3in,paperwidth=8.5in,paperheight=11in]{geometry}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}

\usepackage{dingbat}

\usepackage{amstext}
\usepackage{amsmath, amssymb}

\usepackage{setspace}

\usepackage{xcolor}
\usepackage[colorlinks=true, allcolors=blue!50!black]{hyperref}

\usepackage{titlesec}
\titlespacing\section{0em}{-.25em}{0em}
\titlespacing\subsection{0em}{-.25em}{0em}
\titlespacing\subsubsection{0em}{-.25em}{0em}

\usepackage{color}
\definecolor{lightergray}{rgb}{0.9, 0.9, 0.9}
\usepackage{listings}
\lstset{ %
language=C++,                % choose the language of the code
basicstyle=\footnotesize,       % the size of the fonts that are used for the code
numbers=left,                   % where to put the line-numbers
numberstyle=\footnotesize,      % the size of the fonts that are used for the line-numbers
stepnumber=1,                   % the step between two line-numbers. If it is 1 each line will be numbered
numbersep=5pt,                  % how far the line-numbers are from the code
backgroundcolor=\color{lightergray},  % choose the background color. You must add \usepackage{color}
showspaces=false,               % show spaces adding particular underscores
showstringspaces=false,         % underline spaces within strings
showtabs=false,                 % show tabs within strings adding particular underscores
frame=single,           % adds a frame around the code
tabsize=2,          % sets default tabsize to 2 spaces
captionpos=b,           % sets the caption-position to bottom
breaklines=true,        % sets automatic line breaking
breakatwhitespace=false,    % sets if automatic breaks should only happen at whitespace
escapeinside={\%*}{*)}%,         % if you want to add a comment within your code
%morecomment=[l][]{\#}
}

\title{Programming Choice Experiments in Qualtrics}
\author{Matthew H. Graham}
\date{\today}

\begin{document}

\maketitle

\setlength\parindent{0em}
\setlength\parskip{1em}

This document explains how to program a choice or conjoint experiment in Qualtrics. The approach is designed with two features in mind:  (1) easy customization and (2) no dependence on remote servers to conduct the randomization.

The document's first two sections present step-by-step instructions on a simple setup in which all choice attributes are assigned with equal probability. The third section describes methods of customizing the randomization scheme. The final section presents R code for preparing the resulting Qualtrics dataset for analysis.

The running example will be a choice between two randomly-generated sandwiches:

~

\begin{figure}[h]
\centering
\includegraphics{figures/example.jpg}
\caption{Screen shot of conjoint task.}\label{fig:screen}

\end{figure}

~

This is the first version of this document. If it doesn't explain how to do something you want to do, please contact me at \texttt{matthew.graham@yale.edu}. I'll do my best to assist and will consider adding an answer to your question to the document.

\newpage

\setlength\parskip{0.2em}

\tableofcontents

\newpage


\titleformat{\section}{\singlespacing}{\bfseries\large\thesection.}{0.5em}{\LARGE\bfseries} 
\titleformat{\subsection}{\singlespacing}{{\footnotesize STEP} \thesubsection.}{0.35em}{\large \itshape} 

\newpage

\setlength\parskip{1em}

\section{Randomize}

\subsection{Choose a question early in the survey.}

Start by choosing question that appears somewhere before the conjoint in the survey flow. It can be any question that appears to all respondents (e.g., the consent form, a demographic variable that is asked of all respondents). All that matters is that the randomization code comes before the conjoint experiment.

\subsection{Find the JavaScript console.}

In the current version of Qualtrics, click the settings icon next to the question text, then choose the ``Add JavaScript..." option.

\begin{figure}[h]
\centering
\includegraphics{figures/find_console.jpg}
\caption{Location of the JavaScript console.}

~

\end{figure}

By default, Qualtrics pre-populates some functions: that are needed to add your JavaScript code to the survey. All of the JavaScript for the randomization goes inside the \texttt{Qualtrics.SurveyEngine.addOnload()} function, as indicated below:

\begin{figure}[h]
\centering
\includegraphics{figures/console.jpg}
\caption{Default code appearing in the JavaScript console.}
\end{figure}

\newpage


\subsection{Add the randomization code.}

The following code performs all randomization for ten choices between randomly-generated sandwiches. We'll first examine how this code works, then how to customize it for other applications.

\begin{figure}[h]
\tt
\begin{lstlisting}
Qualtrics.SurveyEngine.addOnload(function(){
	//  Set number of choices;
	var numChoice = 5;
	
	//  Vectors containing all attribute levels:
	var breadArray  = ["Bagel", "Hero", "Roll", "Sliced white", "Tortilla", "Lettuce wrap"];
	var cheeseArray = ["Cheddar", "Gouda", "Jack", "Mozzarella", "Provolone", "None"];
	var greenArray  = ["Arugala", "Green lettuce", "Red lettuce", "Spinach"];
	var meatArray   = ["Ham", "Roast beef", "Turkey", "Portobello", "Egg", "Bean patty"];
	var sauceArray  = ["Hot sauce", "Mayonnaise", "Mustard", "Oil and vinegar", "None"];
	var veggieArray = ["Tomato", "Jalapenos", "Roasted peppers", "Onion", "Olives", "Bean sprouts", "Pickles", "Avocado"];

	//  Fisher-Yates shuffle:
	function shuffle(array){
		for (var i = array.length - 1; i > 0; i--){
			var j = Math.floor(Math.random() * (i + 1));
			var temp = array[i];
			array[i] = array[j];
			array[j] = temp;
		}
		return array;
	}
	
	//  Shuffle a vector, choose the first entry:
	function shuffle_one(theArray){
		var out = shuffle(theArray);
		var out = out[0];
		return(out)
	};
	
	//  Perform the randomization and save the result:
	for(i = 1; i <= numChoice; i++){
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_bread1",  shuffle_one(breadArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_bread2",  shuffle_one(breadArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_cheese1", shuffle_one(cheeseArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_cheese2", shuffle_one(cheeseArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_green1",  shuffle_one(greenArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_green2",  shuffle_one(greenArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_meat1",   shuffle_one(meatArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_meat2",   shuffle_one(meatArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_sauce1",  shuffle_one(sauceArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_sauce2",  shuffle_one(sauceArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_veggie1", shuffle_one(veggieArray));
		Qualtrics.SurveyEngine.setEmbeddedData("choice"+i+"_veggie2", shuffle_one(veggieArray));
	}
});
\end{lstlisting}

\vspace{-.5\baselineskip}

\caption{Randomization JavaScript: fixed attribute order.}\label{fig:randomization_basic}

~

\end{figure}

We want to randomly assign six attributes: bread, cheese, meat (and other hearty things one builds sandwiches around), greens, other vegetables, and sauce. The top of the code block, lines 4-9, creates a vector of levels for each of these six attributes. 

We want to randomly select exactly one of these levels for each sandwich. The middle of the code block contains two functions for doing this. The first, \texttt{shuffle()}, shuffles the vector by implementing the Fisher-Yates shuffle. The second, \texttt{shuffle\_one()}, is a wrapper that runs \texttt{shuffle()} and chooses only the first element. For example, \texttt{shuffle\_one(breadArray)} chooses a random bread.

The final section of the code randomizes and saves the attributes. To embed the randomized variable into the survey, we'll use the function \texttt{Qualtrics.SurveyEngine.setEmbeddedData()}. It takes two arguments: first the new variable's name, then its value. For example, to assign a bagel to the first sandwich in the first choice, one would write

\vspace{.5\baselineskip}

\bgroup
\tt
\begin{lstlisting}
Qualtrics.SurveyEngine.setEmbeddedData("choice1_bread1", "Bagel");
\end{lstlisting}
\egroup

To do this more efficiently, the code loops over all of the sandwiches. Lines 30 through 43 contain a JavaScript for loop. Rather than \texttt{"choice1\_bread1"}, lines 31 through 42 of the code use the formulation \texttt{"choice"+i+"\_bread1"} to automatically do the same randomization for every choice. 

\subsection{Adapt the code to your application.}

To adapt the code to your own purpose, begin by setting the number of choice tasks (line 3) and the changing the attribute levels (lines 6-11). There is no requirement that each attribute have the same number of levels, or limit on the number of levels. If you change the names of the arrays --- you probably should --- don't forget to make corresponding changes in lines 33-44.

The shuffle functions on lines 14-29 do not need to be edited, unless you want to change the randomization scheme. 

The names of the embedded variables that you set in lines 33-44 will eventually appear in your data. For example, in line 33, the text \texttt{"choice"+i+"\_bread1"}  means that the variables \texttt{choice1\_bread1}, \texttt{choice2\_bread1}, ..., \texttt{choice5\_bread1} will be created. You can change this format to match your preferences. Just remember to carry it all the way through in subsequent steps.

Plenty more can be customized within this framework, but this is all you need to do to if you'd like the attributes to display in a fixed order and for each to be drawn with equal probability. Several options for altering the randomization scheme are discussed in section~\ref{sec:custom}.

However you adapt the code at this stage, do so knowing that the naming conventions you choose for your embedded variables will end up affecting how you reshape your data for analysis. In section~\ref{sec:reshape}, the \texttt{R} code takes advantage of features of this naming convention.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage

\subsection{Store the results of your randomization.}

Above, the attributes for all of the conjoint tasks are saved using the \texttt{Qualtrics.SurveyEngine.setEmbeddedData()} function on lines 31-42. This is enough to display these variables later in your survey. However, in order to have the levels appear in your data, you need to take one additional step. \textbf{If you forget to do this, everything will still seem fine in Qualtrics but you won't be able to analyze the data.}

To make sure the variables appear in your data, click the "survey flow" link in the upper-left corner of your Qualtrics screen. 

Once you are in the survey flow, add a new embedded data element to it by clicking "add below" or "add a new element here." Click and drag this element so that it is placed near the beginning of your survey. Crucially, it must be \textbf{above} the question block that includes your randomization JavaScript. I usually put these at the very top of my survey flow.

In your set of embedded data fields, enter the same names from the for loop in lines 30-42 of the code above. For the first pair of sandwiches, the variables look like Figure~\ref{fig:survey_flow}. Leave the values blank, or write something like ``error'' in them if you prefer. They'll be filled in when the respondent reaches the survey question that contains your randomization code.

\begin{figure}[h]
\centering
\includegraphics{figures/embeds.jpg}
\caption{Embedded data fields in the survey flow.}\label{fig:survey_flow}

~

\end{figure}

This must be repeated for every conjoint task. For example, if there are 10 tasks you would need to repeat this block 10 times, changing the variable names to begin with choice2, choice3, etc. This may be the most tiresome step. It goes a little faster if you create all of the variables for one choice or attribute, then duplicate the block and get in a rhythm changing them. But this is step of the process is way too manual for my taste. If you know a better way, please contact me. 

\clearpage
\newpage

\section{Display}

Now that the randomization is finished, the choices need to be displayed. We will make use of a Qualtrics feature called loop and merge. Loop and merge makes it easy to display a series of identically formatted questions.

\subsection{Create a loop and merge block.}

Loop and merge repeats blocks of a Qualtrics survey over and over again, changing only the variables that appear in a table on the back end. In the loop and merge table, each column is a variable and each row is a ``loop'' through the block. For our purposes, ``loop'' is a funny term because the block is only one screen long. To take advantage of the loop and merge feature, we'll need to create a table that has the names of all the embedded variables we created above. 

To start, create a new block and place it where you want your choices to appear in the survey. Open the block options menu and select loop and merge, as shown below.

\begin{figure}[h]
\centering
\includegraphics{figures/loop_merge_where.jpg}
\caption{Location of the loop and merge option.}

~

\end{figure}



\clearpage
\newpage


\subsection{Populate the loop and merge fields.}

A pop-up containing the loop and merge table will appear (Figure~\ref{fig:loop_merge_screen}). Fill in the names of your embedded variables using the following syntax: \texttt{\$\{e://Field/variable\}}. In place of \texttt{variable}, you'll use whatever syntax you used in lines 33-44 of the JavaScript in Figure~\ref{fig:randomization_basic}.

Each row of the table is a choice/conjoint task. This is why all of the entries in the first row of Figure~\ref{fig:loop_merge_screen} begin with ``choice1.'' Each column is an attribute-option combination. In the first column, the suffix ``bread1'' indicates that it is the bread for sandwich option \#1, while the suffix ``bread2'' indicates that it is the bread for sandwich option \#2. 

\begin{figure}[h]
\centering
\includegraphics{figures/loop_merge_backup.jpg}
\caption{Entries in the loop and merge table.}\label{fig:loop_merge_screen}
\end{figure}

\vspace{.5\baselineskip}

The placement of all the ``choice1'' variables in the first row, ``choice2'' in the second row, etc. is very intentional. Alignment between these variables and the loop and merge table will make it easier to prepare your data for analysis.

To fill in the loop and merge table, it may help to create a spreadsheet in Excel or Google Sheets that automatically creates your grid of variable names. Then you can simply paste that grid into the loop and merge interface. 

\begin{figure}[h]
\centering
\includegraphics{figures/loop_merge_grid.jpg}
\caption{Using spreadsheet software to create a grid of embedded variable names.}\label{fig:loop_merge_grid}
\end{figure}

\vspace{.5\baselineskip}

Once you've filled in the loop and merge fields, Qualtrics will automatically run through the question once for every row in the table, regardless of whether you actually use the variables. Below, we'll use the syntax \texttt{\$\{lm://Field/1\}} to display the values entered in the loop and merge table to the respondent.

\clearpage
\newpage

\subsection{Create the question.}

Now we're ready to start displaying the choices to the respondents. To do this, we need two things: a table that displays the choices, and a question that records the choices. The formatting of the table behaves better when the the table and the question are programmed as two separate questions on the same page. 

\begin{figure}[h]
\centering
\includegraphics{figures/separate_Qs.jpg}
\caption{Displaying the table and the response options as two separate questions within the same loop and merge block.}\label{fig:separate_Qs}
\end{figure}

\vspace{.5\baselineskip}

The following describes how to code up the table in HTML. You can also use the Qualtrics rich text editor or create a table in some other software and paste it into Qualtrics and see how it handles the formatting. HTML is better because it's easy to create a cleanly-formatted table that behaves exactly as you tell it to. If you can handle the statistical software you'll use to analyze your results, you shouldn't have much trouble figuring this out.

The code for a basic HTML table is very simple. Start the table with \texttt{<table>} and end it with \texttt{</table>}. Within your table, start each row with \texttt{<tr>} and end it with \texttt{</tr>}. Within each row, start a cell with \texttt{<td>} and end it with \texttt{</td>}. Here we want a 3 x 7 tables, so we'll have three \texttt{<td></td>} pairs nested inside seven \texttt{<tr></tr>} pairs. This block of code creates the table displayed in Figure~\ref{fig:separate_Qs}:

\begin{figure}[h]
\tt
\begin{lstlisting}
[Place any introductory text here.]

<center>
<table>
  <tr><td></td>       <td><b>Sandwich 1</b></td> <td><b>Sandwich 2</b></td></tr>
  <tr><td>Bread</td>  <td>${lm://Field/1}</td>   <td>${lm://Field/2}</td></tr>
  <tr><td>Meat</td>   <td>${lm://Field/3}</td>   <td>${lm://Field/4}</td></tr>
  <tr><td>Cheese</td> <td>${lm://Field/5}</td>   <td>${lm://Field/6}</td></tr>
  <tr><td>Veggie</td> <td>${lm://Field/7}</td>   <td>${lm://Field/8}</td></tr>
  <tr><td>Greens</td> <td>${lm://Field/9}</td>   <td>${lm://Field/10}</td></tr>
  <tr><td>Sauce</td>  <td>${lm://Field/11}</td>  <td>${lm://Field/12}</td></tr>
</table>
</center>
\end{lstlisting}
\vspace{-.5\baselineskip}

\caption{HTML table, unformatted.}\label{fig:HTML_basic}

~

\end{figure}

To add this code to a question, just click the ``HTML View'' tab inside the text editor. It's better to edit your HTML in a separate program like NotePad, then paste it into the HTML view tab each time you want to make a change. This is because Qualtrics automatically adds a bunch of unnecessary tags that will clutter up your code.

This code doesn't set any of its own formatting, other than to center the table (the \texttt{<center>} tags) and make the column headers bold (the \texttt{<b>} tags). In practice, the appearance of the resulting table will depend on your survey's look and feel settings and the respondent's web browser.

\subsection{Change the appearance of your table.}

To make the table look more professional, we'll take advantage of cascading style sheets (CSS), a simple syntax for editing the appearance of elements in a web page. Again, you can accomplish some of the same things using the Qualtrics rich text editor, but CSS is a fairly straightforward way to get a lot more control.

There are two ways to add CSS to an HTML table. First, you can hard-code the style of each element. For example, the tag \texttt{<td style = "text-align:center">} will center the text within your cell. Second, you can create a style sheet that defines classes that you can use over and over again. For example, below we'll define a class called normalrow, then use the tag \texttt{<tr class = "normalrow">} to apply the same formatting to as many rows as we want.

Adding CSS to your table requires two steps. First, open the ``Look \& Feel" menu, then the ``Style" tab. Inside the box labelled ``Custom CSS," insert the code that appears in Figure~\ref{fig:CSS}. 

Second, replace the basic HTML code from Figure~\ref{fig:HTML_basic} with the block of HTML code appearing below in Figure~\ref{fig:HTML_full}. The code is identical, with the exception that each element of the table has now been assigned a \texttt{class}. Line 2 also has an example of how to hard-code a style with \texttt{style}.

If you're happy with the appearance of the table as it appears in Figure~\ref{fig:screen}, all you need to do to customize the table is change the names of the row and column headers in lines 5, 6, 9, 14, 18, 24, 29, and 34. 

If you also want to change the appearance of the table, you'll need to edit the style sheet. As you can see in Figure~\ref{fig:CSS}, CSS syntax is quite intuitive. The comprehensive guide available on \href{https://www.w3schools.com/css/default.asp}{w3schools.com} is a good resource.

\begin{figure}[h]
\tt
\begin{lstlisting}
.wholetable{
  column-gap: 10px; 
  padding: 2%;
  border-left: none;
  border-top: none; 
  border-bottom: solid;
  border-collapse:collapse;
  vertical-align: top;
  text-align: center;
}
.toprow {
  border-bottom: 2px solid #000000 ;  
}
.normalrow{
  vertical-align: top;
  border-bottom: 1px solid #ddd;
}
.bottomrow {
  border-bottom: 2px solid #000000 ;  
}
.normalrow:hover{
  background-color: #f5f5f5 ;
}
.bottomrow:hover{
  background-color: #f5f5f5 ;
}
.attr {
  padding-left: 5%;
  padding-right: 2%;
  padding-top: 1%;
  padding-bottom: 1%;
}
.name {
  padding-left: 2%;
  padding-right: 2%;
  padding-top: 1%;
  padding-bottom: 1%;
  text-align: left;
}
\end{lstlisting}
\vspace{-.5\baselineskip}

\caption{A custom style sheet.}\label{fig:CSS}
\end{figure}


\begin{figure}[h]
\tt
\begin{lstlisting}
<center>
<table class="wholetable" style="min-width:60%">
  <tr class="toprow">
    <td></td>
    <td class="attr"><b>Sandwich 1</b></td>
    <td class="attr"><b>Sandwich 2</b></td>
  </tr>
  <tr class="normalrow">
    <td class="name">Bread</td>
    <td class="attr">${lm://Field/1}</td>
    <td class="attr">${lm://Field/2}</td>
  </tr>
  <tr class="normalrow">
    <td class="name">Meat</td>
    <td class="attr">${lm://Field/3}</td>
    <td class="attr">${lm://Field/4}</td>
  </tr>
  <tr class="normalrow">
    <td class="name">Cheese</td>
    <td class="attr">${lm://Field/5}</td>
    <td class="attr">${lm://Field/6}</td>
  </tr>
  <tr class="normalrow">
    <td class="name">Veggie</td>
    <td class="attr">${lm://Field/7}</td>
    <td class="attr">${lm://Field/8}</td>
  </tr>
  <tr class="normalrow">
    <td class="name">Greens</td>
    <td class="attr">${lm://Field/9}</td>
    <td class="attr">${lm://Field/10}</td>
  </tr>
  <tr class="normalrow">
    <td class="name">Sauce</td>
    <td class="attr">${lm://Field/11}</td>
    <td class="attr">${lm://Field/12}</td>
  </tr>
</table>
</center>
\end{lstlisting}
\vspace{-.5\baselineskip}

\caption{HTML table, formatted.}\label{fig:HTML_full}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\clearpage
\newpage

\titleformat{\subsection}{\singlespacing}{\thesubsection.}{0.35em}{\large \itshape} 

\section{Customize\label{sec:custom}}

Within the architecture laid out so far, it is possible to program the survey to execute a wide range of randomization procedures. This section describes several options for doing so.

\subsection{Randomize the attribute order.}

In some applications, researchers may want to randomize the order in which the choice attributes appear to the respondents. Doing this requires two key steps: modifying lines 30-45 of the randomization JavaScript displayed in Figure~\ref{fig:randomization_basic}, and adopting a new naming convention for the embedded variables.

\begin{figure}[h]
\tt
\begin{lstlisting}
var attribNames = ["Bread", "Cheese", "Greens", "Meat", "Sauce", "Veggie"];
				
function randomize(i){	
	//  Randomly draw the attributes for each sandwich
	var s1 = [shuffle_one(breadArray), shuffle_one(cheeseArray), shuffle_one(greenArray), shuffle_one(meatArray), shuffle_one(sauceArray), shuffle_one(veggieArray)]; 
	var s2 = [shuffle_one(breadArray), shuffle_one(cheeseArray), shuffle_one(greenArray), shuffle_one(meatArray), shuffle_one(sauceArray), shuffle_one(veggieArray)];
		
	//  Randomize the display order			
	var attribOrder = shuffle([1,2,3,4,5,6]);
			
	//  Save the results
	for(var k = 1; k <= attribOrder.length; k++){
		var index = attribOrder[k-1]-1;
		Qualtrics.SurveyEngine.setEmbeddedData('c'+i+'_attrib'+k+'_name', attribNames[index]);
		Qualtrics.SurveyEngine.setEmbeddedData('c'+i+'_attrib'+k+'_sand1', s1[index]);
		Qualtrics.SurveyEngine.setEmbeddedData('c'+i+'_attrib'+k+'_sand2', s2[index]);
	};
};
				
for(var i=1; i <= numChoice; i++){
	randomize(i);
};
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Randomization JavaScript: random attribute order.}\label{fig:randomization_attributes}
	

\vspace{.5\baselineskip}
	
\end{figure}

To account for the fact that the attributes (bread, meat, cheese, etc.) are going to appear in random order, we need to modify the naming convention for the attributes, which had previously hard-coded bread, cheese, etc. Figure~\ref{fig:randomization_attributes} implements an alternate naming convention this in lines 14-16. For the first choice, the first attribute of sandwich 1 is \texttt{c1\_attrib1\_sand1}. The first attribute of sandwich 2 is \texttt{c1\_attrib1\_sand2}. There is also a new set of fields, \texttt{c1\_attrib1\_name}, that contain the names of the attributes. 

You'll also need to use this variable naming convention in your survey flow (Figure~\ref{fig:survey_flow}) and your loop and merge table (Figure~\ref{fig:loop_merge_screen}). For example, here is how the first four columns of your loop and merge table would look with this modified naming conventions:

\begin{figure}[h]
\centering
\includegraphics{figures/loop_merge_table_random_order.jpg}
\caption{Loop and merge table with naming convention for randomized attribute order.}

\vspace{-2\baselineskip}

\end{figure}


\subsection{Randomize with unequal probability.}

So far, all of the attributes have been assigned with equal probability. We've accomplished this by creating vectors, shuffling them, and choosing the first element. Researchers sometimes want to randomize attributes with unequal probability. To do this for bread, we can create a new function called \texttt{return\_bread()} to carry out a different randomization scheme. Then we'll replace of all the instances of \texttt{shuffle\_one(breadArray)} with \texttt{return\_bread()}. 

Suppose, for whatever reason, that we would like 40 percent of our sandwiches to be lettuce wraps, 20 percent to be tortilla wraps, and 10 percent each to come from the other four categories of bread. The following function returns a type of bread with these probabilities:
\begin{figure}[h]
\tt
\begin{lstlisting}
function return_bread(){	
	var num = Math.random();
	if(num < .4){var out = "Lettuce wrap"
		} else if(num < .6){var out = "Tortilla"
		} else if(num < .7){var out = "Bagel"
		} else if(num < .8){var out = "Hero"
		} else if(num < .9){var out = "Roll"
		} else {var out = "Sliced white"};
	return out
};
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Function to choose attributes with unequal probability.}\label{fig:randomization_unequal}

\vspace{-1\baselineskip}
	
\end{figure}

The function \texttt{return\_bread()} uses \texttt{Math.random()} to draw a random number between 0 and 1, then uses the \texttt{if} statements to specify the desired CDF of the randomization distribution.

\newpage

\subsection{Randomly choose from a range of integers.}

Researchers sometimes want to assign numerical values as attributes. This can be accomplished using the tools above by simply creating a vector containing all of the numerical values. It can also be accomplished by writing a specialized function to return a number.

Departing from the sandwich example, suppose that we would like to assign a fake candidate an age between 40 and 65. The following function returns these integers with equal probability. It can be accommodated into the code in the same way as the \texttt{return\_bread()} function above.

\begin{figure}[h]
\tt
\begin{lstlisting}
function return_age(){
	var out = Math.floor(Math.random()*26) + 40;
	return out
};
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Function to draw a random integer between 40 and 65.}\label{fig:randomization_unequal}
	
\end{figure}

The function \texttt{return\_age()} produces a random integer between 40 and 65 by drawing a random number between [0,1) using \texttt{Math.random()}, multiplying it by 26 (i.e., the desired number of integers), rounding it down with \texttt{Math.floor()}, and adding 40.

Making this function more flexible adds only a bit of complexity. Suppose we want to be able to use the same function to generate multiple integer ranges. We can do this as follows:

\begin{figure}[h]
\tt
\begin{lstlisting}
function return_integer(low, high){
	var multiplier = high - low + 1;
	var out = Math.floor(Math.random()*multiplier) + low;
	return out
};
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Function to draw a random integer between specified values.}\label{fig:randomization_unequal}
	
\end{figure}

In \texttt{return\_integer()}, the variables \texttt{low} and \texttt{high} play the roles that 40 and 65 played in \texttt{return\_age()}.

\newpage

\subsection{Randomize and record the order in which the choices appear.}

If your choices are all generated using the same randomization scheme, the order in which they appear doesn't really matter. In these cases, it's better not to randomize the order of the choices. That way, the choice number that appears in each of your embedded variables already tells you the order in which the choices appeared.

Researchers sometimes want to use different randomization schemes for different choices. In these cases, it's really handy to hard-code the choice number, then randomize the order in which the choices appear. For example, in some of my research, I have coded custom randomization schemes that over-sample more realistic scenarios in a subset of the choices. In these situations, the choice number tells me which randomization scheme was used, and randomizing the choice order allows me to ensure that choice order isn't correlated with the probability of assignment.

To randomize and record the order of the choices, one can take the following steps:

\vspace{-1\baselineskip}

\begin{enumerate}
\item Inside the loop and merge table, check the "Randomize loop order" box.
\item Add a column to your loop and merge table containing choice number.
\item Add an embedded variable called \texttt{choice\_order} to the survey flow.
\item Add the following JavaScript to one of the slides on the page with the candidate choice:
\end{enumerate}

\vspace{-1\baselineskip}

\begin{figure}[h]
\tt
\begin{lstlisting}
var order =  "${e://Field/choice_order} " + "; " + "${lm://Field/20}"
Qualtrics.SurveyEngine.setEmbeddedData("choice_order", order);
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Recording choice order.}\label{fig:randomization_unequal}
	
\end{figure}
This code works by (1) reading the order of the previous choices out of \texttt{choice\_order}, (2) appending the number assigned to the current choice, and (3) writing the result back onto \texttt{choice\_order}. In place of the 20 in line 1, input the number of the column you created to store the choice numbers.

\newpage

\subsection{Randomize images.}

Randomizing images is easier than you might think. You can insert an image anywhere in your Qualtrics survey using the HTML code \texttt{<img src="url">}. This means that at the end of the day, all you really have to do to randomize an image is replace one the arrays of attribute names at the top of the code (lines 6-11 in Figure~\ref{fig:randomization_basic}) with an array of URLs. Make sure to include the full URL, including the \texttt{http://}.

Though the images can be stored anywhere on the web, the best place to store them is your Qualtrics library, which will be accessible to anyone who can access your survey. One way to retrieve the URL is to (1) create a new graphic question, (2) upload a graphic from your computer or choose one from your library, (3) preview the question, and (4) right click the image to copy the URL. The last step of this process is shown in Figure~\ref{fig:get_url}. You'll get a URL that looks something like \\ \texttt{https://yalesurvey.ca1.qualtrics.com/WRQualtricsControlPanel/Graphic.php?IM=IM\_0e67RxsPjQCPhsh}



\begin{figure}[h]
\centering
\includegraphics{figures/get_image_url.jpg}
\caption{Locating the URL of an image from your library.}\label{fig:get_url}

\vspace{\baselineskip}

\end{figure}

However you choose to host your images, what matters is that you get a set of URLs that corresponds to the images you want to randomize. For example, you could change line 6 in Figure~\ref{fig:randomization_basic} to:

\begin{figure}[h]
\tt
\begin{lstlisting}
var breadArray = ["http://website.com/chart1.jpg", "http://website.com/chart2.jpg", 
    "http://website.com/chart3.jpg", "http://website.com/chart4.jpg"]
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{An array of image URLs.}\label{fig:randomization_unequal}
	
\vspace{\baselineskip}
\end{figure}

You can then save these URLs in the same way you saved text before. You can also put complete HTML tags inside of embedded variables if you prefer.

The only other thing that's different is adding the image to your display. For example, if we wanted to change the cell entries in the top row of Figure~\ref{fig:screen} to pictures of bread, we could replace lines 9-12 of Figure~\ref{fig:HTML_full} with

\begin{figure}[h]
\tt
\begin{lstlisting}
<tr class="normalrow">
  <td class="name">Bread</td>
  <td class="attr"><center><img src="${lm://Field/1}" width=90%></center></td>
  <td class="attr"><center><img src="${lm://Field/2}" width=90%></center></td>
</tr>
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Adding images to a row of the choice table.}\label{fig:add_image}

  \end{figure}

The code in Figure~\ref{fig:add_image} displays the two images centered within their cells. Each image is set to take up 90 percent of the cell's width. You can also set this using CSS with the syntax \texttt{<img src="url" style="width:90\%"}. Setting image width as a percentage is a good practice for surveys because it re-sizes well.

~

Of course, you probably don't want to put a picture of bread into a row in your choice table.  For example, you might want to put a person's face above each column. To accomplish this, you'd want to put a new row at the top of your table and style it in a way that suits your needs. To add an image at the top of each column, centered and with 10 pixels of padding below each image, one would use the following code:
\begin{figure}[h]
\tt
\begin{lstlisting}
<tr style="padding-bottom:10px; text-align:center">
  <td></td>
  <td><img src="${lm://Field/1}" width=90%></td>
  <td><img src="${lm://Field/2}" width=90%></td>
</tr>
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Adding images to a row of the choice table and specifying a different style for that row.}

  \end{figure}

Ultimately, you have a lot of flexibility in terms of how you format a table, or images within it. The images don't have to be in the table, either.

In general, the only tricky part about inserting images via your own HTML tags is giving the page good instructions about how to deal with variable screen widths. It can be helpful to supplement the \texttt{width} with the \texttt{min-width} or \texttt{max-width} CSS attribute. For example, suppose you want an image to shrink to fit small screens but not expand beyond the image file's actual width, 500 pixels. The code \texttt{<img src="url" style="width:100\%; max-width:500px">} accomplishes this. If you wanted to apply the same rule to a lot of images, you could create a class and add it to your custom CSS file (Figure~\ref{fig:CSS}).

%  DUMPING GROUND FOR UNWRITTEN SECTION IDEAS

%\subsection{Randomize images or names and associate them with particular attributes.}

%\subsection{Reformat the table to better-accommodate verbose attribute levels.}

%[Fix the number of times a respondent sees an attribute]
%
%[Fix the combinations of attributes]

%\subsubsection{Randomizing which option appears on the left or right.}

%[I prefer for candidate 1 to be hard-coded when I collect my data, then randomize the side.]

%\subsubsection{Using apostrophes and quotation marks.}

%\subsection{Draw multiple attributes from a category.}

%\subsection{Randomize and record the side on which the choices appear.}
%
%If your choices are all generated using the same randomization scheme, the side on which they appear doesn't really matter. However, researchers sometimes have a theoretical reason to randomize the two options differently. For example, one might want to specify that one option is always a nuclear weapon and the other is always a conventional weapon. In this case, it can be handy to designate 


\newpage

\subsection{Use vignettes instead of tables.}

Conjoint experiments are typically presented as tables, but the same machinery can be used to randomly generate variables in a vignette experiment. For example, one could replace the HTML table with the following:

\begin{figure}[h]
\tt
\begin{lstlisting}
${lm://Field/3} and ${lm://Field/5} on ${lm://Field/1} with ${lm://Field/7}, ${lm://Field/9}, and ${lm://Field/11}.
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Using the same randomization code to program a vignette.}\label{fig:randomization_unequal}
	
\end{figure}

\vspace{.5\baselineskip}

Researchers running vignette experiments often prefer to show only one vignette to each respondent. If so, you can skip the loop and merge business altogether and directly program in embedded fields, like so:

\begin{figure}[h]
\tt
\begin{lstlisting}
${e://Field/meat} and ${e://Field/cheese} on ${e://Field/bread} with ${e://Field/greens}, ${e://Field/veggie}, and ${e://Field/sauce}.
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Using the same randomization code to program a vignette with no loop and merge fields.}\label{fig:randomization_unequal}
	
\end{figure}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%55
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage

\section{Analyze}

When you download your data from Qualtrics, each row will represent a respondent. You probably want your unit of analysis to be either a sandwich or a choice between two sandwiches. This section describes how to prepare your data for analysis.

The code here uses the \texttt{R} statistical programming language. It requires the \texttt{dplyr} and \texttt{magrittr} packages, both of which are installed as part of the \texttt{tidyverse} package.

\subsection{Prepare the data.}

Reshaping the data will require three sets of variables: a unique respondent identifier, the variables containing the characteristics, and the variables containing the respondents' choices. To keep things simple, let's assume you have reduced your data to contain only these variables. We'll also keep the examples a bit more concise by getting rid of the sauce and the greens.

If you follow the suggested naming conventions at the randomization and display stages, your dataset will look something like this:
\begin{figure}[h]
\tt
\begin{lstlisting}
   id choice1_bread1 ... choice5_veggie2     1_pref     2_pref     3_pref ...
1   1   Sliced white     Roasted peppers Sandwich 2 Sandwich 2 Sandwich 1    
2   2           Roll             Avocado Sandwich 1 Sandwich 1 Sandwich 2    
3   3           Roll               Onion Sandwich 1 Sandwich 1 Sandwich 2    
4   4          Bagel             Avocado Sandwich 2 Sandwich 1 Sandwich 2    
5   5       Tortilla               Onion Sandwich 1 Sandwich 1 Sandwich 1    
6   6           Hero             Avocado Sandwich 1 Sandwich 1 Sandwich 2    
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Data structure.}\label{fig:data}
	
\end{figure}

\texttt{id} is the respondent identifier. You can create this yourself or use the identifiers supplied by Qualtrics or your vendor. Wherever \texttt{id} comes from, make sure it takes a unique value in each row and that there are no missing values. Duplicates or missingness in \texttt{id} will mess up the code below.

The variables prefixed \texttt{choice} contain the sandwich attributes that were displayed to the respondent. These follow the naming scheme established in the randomization JavaScript and carried through to the display table.

The variables prefixed with numbers contain the respondents' preferences. Loop and merge blocks automatically attach these prefixes, e.g., \texttt{1\_pref} is the first instance of the question called ``pref.'' If you set up your loop and merge table so that variables prefixed \texttt{choice1} always appear in the first row, etc., then the numbering of the \texttt{choice} and \texttt{pref} variables will be aligned. The code below assumes that the numbers align.

\newpage

\subsection{Reshape the data.}

There's more than one way to reshape the data. Figure~\ref{fig:Rcode} shows one good way to do it in three steps. It assumes that the \texttt{data.frame} printed in Figure~\ref{fig:data} is stored as an object called \texttt{d}.

\begin{figure}[h]
\tt
\footnotesize
\begin{lstlisting}
#  Reshape the sandwich attributes so each sandwich gets its own row.
d_sand = d %>% select(id, starts_with("choice"))
d_sand = d_sand %>%
  gather(variable, value, -id) %>%
  mutate(
    choiceNum = gsub("[A-Za-z]|_.+", "", variable),
    sandNum   = gsub(".+(.$)", "\\1", variable),
    attribute = gsub(".+_|.$", "", variable)
  ) %>%
  select(-variable) %>%
  spread(attribute, value)

#  Reshape the respondent's preferences so each choice gets its own row.
d_pref = d %>% select(id, ends_with("pref"))
d_pref = d_pref %>%
  gather(variable, preference, -id) %>%
  mutate(
    choiceNum  = gsub("_pref", "", variable),
    preference = as.numeric(gsub("Sandwich ", "", preference))
  ) %>%
  select(-variable)

#  Merge the attributes and preferences.
d_stack = left_join(d_sand, d_pref)
d_stack = d_stack %>%
  mutate(
    Y = as.numeric(sandNum == preference)
  )

#  Check that you did not create any extra rows.
nrow(d_stack) == (nrow(d) * max(d_stack$sandNum) * max(d_stack$choiceNum))
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{\texttt{R} code for reshaping the data.}\label{fig:Rcode}

~
	
\end{figure}

The first step is to split off the sandwich attributes and turn these into a dataset in which each row corresponds to a sandwich. To begin, line 2 splits off \texttt{id} and the choice attributes. Next, line 4 stacks the data so that each sandwich $\times$ characteristic combination gets its own row. Using regular expressions, lines 5-9 break each variable name into its component parts: the choice number (in our example before, 1 to 5), the sandwich number (1 or 2), and the attribute name. Finally, line 10 drops the original variable name and line 11 spreads the attributes back out into the columns. This yields a \texttt{data.frame} with the following structure:

\begin{figure}[h]
\tt
\begin{lstlisting}
    id choiceNum sandNum        bread     cheese       meat          veggie
1    1         1       1 Sliced white       Jack        Ham          Tomato
2    1         1       2     Tortilla Mozzarella Roast beef    Bean sprouts
3    1         2       1        Bagel       Jack     Turkey Roasted peppers
4    1         2       2 Lettuce wrap Mozzarella Portobello    Bean sprouts
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Structure of \texttt{d\_sand} as of line 11 in Figure~\ref{fig:Rcode}.}\label{fig:d_sand}
	
~

\end{figure}

%If you put the attributes in random order, you'll need to adapt lines 3-11.

The second step is to split off the respondent's preferences and turn these into a dataset in which each row corresponds to a choice between two sandwiches. To begin, line 14 splits off \texttt{id} and the respondent's preferences. Line 16 stacks the data, line 18 removes the \texttt{\_pref} suffix from each variable name and stores the prefix as the choice number, and line 19 removes the word ``Sandwich'' from the variables containing the respondent's choices. This yields a \texttt{data.frame} with the following structure:

\begin{figure}[h]
\tt
\begin{lstlisting}
   id choiceNum preference
1   1         1          2
2   2         1          1
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Structure of \texttt{d\_pref} as of line 21 in Figure~\ref{fig:Rcode}.}\label{fig:d_pref}
	
~

\end{figure}

The third step is to join \texttt{d\_sand} and \texttt{d\_pref} together. Comparing Figure~\ref{fig:d_sand} and Figure~\ref{fig:d_pref}, you'll notice two common variables: \texttt{id} and \texttt{choiceNum}. This reflects the structure of the two datasets. \texttt{d\_sand} should have twice as many rows as \texttt{d\_pref}.

Line 24 binds these two datasets together. \texttt{left\_join()} will automatically find the common variables and duplicate the rows so that each entry in \texttt{d\_pref} is matched to both corresponding entries in \texttt{d\_sand}. If you're a purist, you can also use the \texttt{by} argument to tell \texttt{left\_join()} which variables to use. The resulting dataset, \texttt{d\_stack}, is at the sandwich level, just like \texttt{d\_sand}. Line 27 creates a new variable, \texttt{Y}, that indicates whether the respondent preferred that sandwich. This yields a \texttt{data.frame} with the following structure:

\begin{figure}[h]
\tt
\begin{lstlisting}
   id choiceNum sandNum        bread     cheese       meat          veggie preference Y
1   1         1       1 Sliced white       Jack        Ham          Tomato          2 0
2   1         1       2     Tortilla Mozzarella Roast beef    Bean sprouts          2 1
3   2         1       1         Roll       Jack     Turkey          Tomato          1 1
4   2         1       2        Bagel    Cheddar Roast beef    Bean sprouts          1 0
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Structure of \texttt{d\_stack} as of line 28 in Figure~\ref{fig:Rcode}.}\label{fig:d_stack}
	
~

\end{figure}

At the end, line 31 verifies that \texttt{d\_stack} has the correct number of rows. This line returns TRUE if \texttt{d\_stack} has exactly five choices $\times$ two sandwiches per choice = ten times the number of rows as the initial dataset, \texttt{d}. If you are creating extra rows, you probably have duplicates or missingness in \texttt{id}. If you don't have enough rows, some part of your code above probably isn't running due to an error and you haven't noticed yet.

If you plan to use other information about your respondents in the analysis, you can use \texttt{id} and \texttt{left\_join()} to add those variables to \texttt{d\_stack}.

\subsection{Perform analysis.}

In this form, \texttt{d\_stack} is ready for any analysis that takes the sandwich as the unit of observation, e.g. the popular Average Marginal Component Effect (AMCE) framework. Using the \texttt{estimatr} package, you could compute AMCEs as follows:

\begin{figure}[h]
\tt
\begin{lstlisting}
lm_robust(Y ~ bread + cheese + meat + veggie, data = d_stack, clusters = id)
\end{lstlisting}
\vspace{-.5\baselineskip}

	\caption{Calculating the AMCE using \texttt{d\_stack}.}\label{fig:d_stack}
	
~

\end{figure}

Now all you have to do is write the paper!

\end{document}


